perm filename DERIV.TIM[TIM,LSP]16 blob sn#753224 filedate 1984-05-01 generic text, type C, neo UTF8
COMMENT ⊗   VALID 00019 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00003 00002	Run 500. times, benchmark in deriv.mcl[tim,lsp] functionn call is (timit)
C00004 00003	∂16-Oct-81  1037	ROD   via MIT-AI 	DERIV  
C00006 00004	∂15-Oct-81  2343	pratt@Diablo (SuNet) 	Dolphin timings   
C00013 00005	∂15-Oct-81  2110	pratt@Diablo (SuNet) 	Dolphin benchmarks
C00014 00006	∂17-Oct-81  1731	pratt@Diablo (SuNet) 	Dolphin timings   
C00019 00007	 New SAIL
C00022 00008	∂06-Jul-82  1703	Kim.fateman at Berkeley 	Re:  Progress  
C00024 00009	∂06-Jul-82  1802	Kim.fateman at Berkeley 	deriv
C00025 00010	∂16-Jul-82  0012	Mabry Tyson <Tyson at SRI-AI> 	DERIV, DDERIV, FDDERIV results    
C00030 00011	LM-2
C00031 00012	DERIV
C00032 00013	DERIV
C00033 00014	 NIL
C00034 00015	 SCORE dec 12, 1983
C00035 00016	 DEC780CL
C00036 00017	 InterLisp Vax 780
C00037 00018	 PSL-20
C00038 00019	 PSL-Cray
C00039 ENDMK
C⊗;
Run 500. times, benchmark in deriv.mcl[tim,lsp] functionn call is (timit)
Interpreted SAIL MacLisp
(RUNTIME 5.612) 
(GCTIME 0.346) 
T 
(RUNTIME 5.629) 
(GCTIME 0.309) 
T 
(RUNTIME 5.635) 
(GCTIME 0.463) 
T 
Compiled SAIL MacLisp
(RUNTIME 0.591) 
(GCTIME 0.344) 
T 
(RUNTIME 0.588) 
(GCTIME 0.298) 
T 
(RUNTIME 0.587) 
(GCTIME 0.449) 
T 

5000. times, compiled MC MacLisp
(runtime 7.938372)
(gctime 2.58056)
∂16-Oct-81  1037	ROD   via MIT-AI 	DERIV  
To:   RPG at SU-AI, VRP at SU-AI 
I just ran DERIV on  CADR-5. It has 192K of physical memory. I ran  it in
an address space I'd been using all morning. I didn't switch anything off - i.e.
left it listening to the Chaos Net, and running the little "who line" display
at the bottom of the screen. From that display it looked like it was paging madly
as it chewed up cons cells.

Interpreted on 500. callsran it two times - both came in at between 51 and 52 seconds.

Compiled on 5000. calls ran it five times: 50.95, 59.233, 60.766, 59.3, 59.637.

[the first reading seems counter intuitively loer - it had just done the compilation
and loading, and then, I guess switched regions to the default user cons region -
may be some strange behavior of the paging algorithm which happens because the
ratio of consing to computing is so atypically high here.]

Keep in mind that this is an AI lab CADR, not a symbolics A machine, and certainly
not in the class of  a Symbolics 3600. I will try it on a machine with more physical
memory when one is available.

∂16-Oct-81  1052	ROD   via MIT-AI 	P.S.   
To:   RPG at SU-AI, VRP at SU-AI 
They were all wall clock times, not times from internal timers.

∂15-Oct-81  2343	pratt@Diablo (SuNet) 	Dolphin timings   
Date: 15 Oct 1981 23:34:59-PDT
From: pratt at Diablo
To: equip
Subject: Dolphin timings

I have benchmarked two Lisp programs on the Stanford Dolphins.  The result was
an eye-opener for me, but came as no surprise to HPP-er Gordon Novak, who
has ported a large natural language system, ISAAC, from the U. of Texas
2060 Interlisp to Dolphin Interlisp.  Gordon cited a ratio of about 28 
between these two machines for all phases of Interlisp use, namely 
compilation, loading, and execution.  I asked him for an ISAAC demo, so he
proceeded to load his already compiled system, which he told me had taken 50 
minutes to compile on the Dolphin.  We chatted about features of Interlisp to 
while away the 10 minutes it took the Dolphin to load his demo.  He then ran 
ISAAC on a number of physics problems stated in English.  Problem 1 took 45 
seconds.  Gordon said that when running interpreted on the Dolphin this 
problem had taken 180 seconds, as against 6.4 seconds running interpreted on 
the 2060.  He did not have timings for compiled code on the 2060, though 1-2 
seconds is a reasonable guess.  The remaining problems all showed similar 
behavior.

Gordon commented, "This sort of performance really changes your style of using
computers.  You arrange to work on other things while your program is loading,
or to go out for lunch while it is compiling."

My observations were in close agreement with Gordon's, despite my programs
being much smaller and also more CONS-intensive.  One program took derivatives
of algebraic expressions.  Taking the derivative of 3x↑2+ax↑2+ax+5 in Franz
Lisp on Diablo took 3.4 milliseconds, 4.8 milliseconds in a toy Lisp
system on the Sun terminal, and 6.6 milliseconds in UCI-Lisp on Sumex's KI-10.
On the Dolphin it took 160 milliseconds.  (All these timings were obtained by 
running the problem a large number of times, bringing the measured time up to 
the order of many seconds.)

Another program converted logical formulas into disjunctive normal form.  One 
fairly large formula took 21 milliseconds in Franz Lisp, 29 milliseconds on the
Sun, and 1 second on the Dolphin.

Yesterday a contingent from Xerox visited Stanford to discuss Dolphins in
general and Dolphin Interlisp in particular.  One topic was how to port
Interlisp programs to the Dolphin.  Another was a breakdown of Interlisp
performance on the Dolphin; for example CONS was described as taking around 800
milliseconds.  CONS and function-call-and-return were identified as performance
bottlenecks; CAR and CDR however were described as being fast, although no
figures were given for them.  Masinter is optimistic about being able to make
30% or better improvements in the performance of CONS and call-and-return;
meanwhile one should expect particularly bad performance from programs that
make excessive use of CONS and function calls.

One conclusion from all this is that the cost-effective way to get Interlisp
cycles for those users for whom address space is not a problem is to buy a
2060, which gets you the equivalent of nearly 30 Dolphins, not to mention the
advantage of having more than one Dolphin's worth of power on those occasions
when you aren't competing with 30 other Interlisp users.  I readily concede
that users running up against address space limitations have a problem.
However it is far from clear that Dolphins are the solution to that problem for
any but the most patient of users.  If one is to believe Dick Gabriel's
claim that a Vax 11/780 becomes unusable with two Franz Lisp users, one must
also believe that a Dolphin becomes unusable as soon as it has more than 1/25
of an Interlisp user.

The other conclusion is that no one but a dyed-in-the-wool Interlisp enthusiast
is going to make serious use of Interlisp on the Dolphins.  I am a Lisp
enthusiast myself, but not to the extent that I would bother to use a Lisp as
slow as that on the Dolphins.

Symbolics 3600's sound considerably more attractive in the light of these
Dolphin measurements.  I am very anxious to see these measurements taken on the
3600.

			Vaughan

∂15-Oct-81  2110	pratt@Diablo (SuNet) 	Dolphin benchmarks
Date: 15 Oct 1981 21:01:43-PDT
From: pratt at Diablo
To: RPG@SU-AI, VRP@SU-AI
Subject: Dolphin benchmarks

Sorry, haven't put these in a file yet.  DERIV took 800 seconds compiled
against 24 seconds on the Sun for 5000 iterations.  I just timed DNF a few
hours ago and got 3 seconds compiled for three iterations, so 1000 iterations
should be around 1000 seconds as against 29 seconds on the Sun.  Incidentally
compilation yielded a factor of three improvement for DNF.

∂17-Oct-81  1731	pratt@Diablo (SuNet) 	Dolphin timings   
Date: 17 Oct 1981 17:32:02-PDT
From: pratt at Diablo
To: CSD.GENESERETH@SU-SCORE at SUMEX-AIM, CSD.GERRING@SU-SCORE at SUMEX-AIM,
    CSD.HBROWN@SU-SCORE at SUMEX-AIM, CSD.NOVAK@SU-SCORE, CSD.NOVAK@SU-SCORE at SUMEX-AIM,
    RPG@SU-AI, SCHOEN@SUMEX-AIM, equip@SU-HPP-VAX
Subject: Dolphin timings

My apologies for appearing to be bad-mouthing the Dolphin.  I'll try to get my
tests and my remarks both more neutral and more accurate in future.  

Gordon is quite right about not nettling Xerox.  Clearly the goal should be to
constructively encourage Xerox to improve the Dolphin in those areas where 
it hurts the most.  Equally clearly, this need not imply a commitment
on anyone's part to buy Dolphins in significant volume if it is eventually
concluded that they are not sufficiently effective Lisp tools.  Both further 
Interlisp development and further benchmarking (or just plain experience with 
InterlispD), including getting programs adjusted to the Dolphin environment,
are needed before decisions of this sort can be made.

My apologies for getting Gordon's compile-time and load-time ratios wrong, I
misunderstood what he had told me about this.  (The perils of being a
reporter!)  My "800 milliseconds" for CONS was of course a typo for 800
microseconds, .8 seconds for a CONS would be nonsense even on an 8080.  (I 
wonder about that 800 microseconds.  My DERIV program performed 305,000 
conses and took 800 seconds.  If conses take 800 microseconds then that only 
accounts for 250 seconds.  Hence either CONS does NOT dominate the 
computation time of DERIV or CONS takes more like 2.4 milliseconds on the
Stanford Dolphins.)

I agree heartily with Gordon's statement that writing time is more important 
than running time.  That's why I prefer to write in a standard Lisp style,
without spending time worrying about how to transform my programs to eliminate
consing.  If when I use a cons-intensive style my programs slow to a crawl then
I feel justified in lodging a performance complaint.

Gordon's upbeat remarks about using the Dolphin as a Lisp machine (which seemed
much more positive than when I was talking to him on Thursday) would appear
to contradict Dick Gabriel's feeling that a Vax 11/750 would be "unusable" on
performance grounds as a personal Lisp machine.  It would be nice to get some
agreement from the Lisp community as to what level of performance is tolerable
on a personal computer.  (As a Lisp user myself, I lean more to Dick's position
than Gordon's - nice environment notwithstanding, I think I'd find the Dolphin
slower than I could bear.)

	Vaughan

; New SAIL

(fasload deriv)
(timit)

Timing performed on Thursday 05/27/82 at 15:29:08.
Cpu Time = 2.121
Elapsed Time = 119.15
Wholine Time = 38.133333
GC Time = 18.467
Load Average Before = 2.21320271
Load Average After  = 2.36379516
Average Load Average =2.28849894
NIL 

Timing performed on Thursday 05/27/82 at 15:31:20.
Cpu Time = 2.124
Elapsed Time = 63.6666665
Wholine Time = 39.2166667
GC Time = 18.021
Load Average Before = 2.18321395
Load Average After  = 1.90828836
Average Load Average =2.04575115
NIL 

Timing performed on Thursday 05/27/82 at 15:32:43.
Cpu Time = 2.118
Elapsed Time = 118.566667
Wholine Time = 43.3
GC Time = 18.061
Load Average Before = 1.70717573
Load Average After  = 1.97787261
Average Load Average =1.84252417
NIL 

(fasload deriv)
(timit)
Timing performed on Wednesday 10/12/83 at 22:03:24.
Cpu (- GC) Time = 1.808
Elapsed Time = 55.9666667
Wholine Time = 36.4166665
GC Time = 17.113
Load Average Before  = 0.79434204
Load Average After   = 1.02131605
Average Load Average = 0.907829046
NIL 
Timing performed on Wednesday 10/12/83 at 22:04:31.
Cpu (- GC) Time = 1.802
Elapsed Time = 55.0
Wholine Time = 30.2333333
GC Time = 17.113
Load Average Before  = 1.0043279
Load Average After   = 1.26127863
Average Load Average = 1.13280326
NIL 
Timing performed on Wednesday 10/12/83 at 22:05:28.
Cpu (- GC) Time = 1.801
Elapsed Time = 49.2166667
Wholine Time = 31.8
GC Time = 17.114
Load Average Before  = 1.25375867
Load Average After   = 1.35089779
Average Load Average = 1.30232823
NIL 
∂06-Jul-82  1703	Kim.fateman at Berkeley 	Re:  Progress  
Date: 6 Jul 1982 16:49:48-PDT
From: Kim.fateman at Berkeley
To: RPG@SU-AI
Subject: Re:  Progress
Cc: Kim.jkf@Berkeley

I just ran DERIV; Franz Lisp opus 38.20 on a vax 11/780 running with
a load average of between .5 and .8  (light).  Recorded time 27 seconds
of which 16 were in garbage collection.
The only change to the file was to declare (localf der1 deriv).

jkf: get back to your thesis.

∂06-Jul-82  1740	ARPAVAX.fateman at Berkeley   
Date: 6 Jul 1982 17:35:21-PDT
From: ARPAVAX.fateman at Berkeley
To: rpg@su-ai
Cc: ARPAVAX.jkf@Berkeley

jkf reminded me to do (sstatus translink on)  to get faster times,
so with that, on a 780:
deriv = 25,2, 16.9 of which is GC
dderiv = 27.4, 17.6 of which is GC

fdderiv requires subrcall which we don't have exactly.  I haven't
looked to see how funcall can work instead.

∂06-Jul-82  1802	Kim.fateman at Berkeley 	deriv
Date: 6 Jul 1982 17:57:22-PDT
From: Kim.fateman at Berkeley
To: rpg@su-ai
Subject: deriv
Cc: Kim.jkf@Berkeley

for a (load average 4) vax 750, no floating point accelerator, Franz opus 38.20
36.06 of which 22.2 is in GC.

I think a lighter load on the 750 would help somewhat.

∂16-Jul-82  0012	Mabry Tyson <Tyson at SRI-AI> 	DERIV, DDERIV, FDDERIV results    
Date: 16 Jul 1982 0003-PDT
From: Mabry Tyson <Tyson at SRI-AI>
Subject: DERIV, DDERIV, FDDERIV results
To: rpg at SU-AI

Results for DERIV, DDERIV, FDDERIV for UCILISP (UT's version).

All timings are on SRI-AI's 2060 with load average around 0.2.

Notes on coding of the programs:
UCILISP open codes MAPCAR if the function is specified by a
LAMBDA expression but does not if the function is a defined function.
Therefore, (MAPCAR 'FOO BAR) was translated into (MAPCAR (FUNCTION
(LAMBDA (X) (FOO X))) BAR) to keep in the spirit of the open coded
version sent out.

DDERIV and FDDERIV involve calling a function which is the value of
a local variable.  In UCILISP this may be done by simply calling
(FOO ...) where FOO is the local variable rather than fooling with
FUNCALL or SUBRCALL.  This format handles both cases.  With (NOUUO
NIL), these function calls do not become direct jumps (because the
value may change next time).

There was a slight problem in having the compiler produce SUBR code
to be stored under a different property (DERIV).  It could be done
but not conveniently.  So I just compiled the code as SUBR's and
editted the LAP code to make it load onto the DERIV property.

DDERIV and FDDERIV were the version that had the (CONS 'TIMES A) in
the definition for TIMES.

The loop was a PROG which had 5 calls to DERIV and looped 1000 times.
The function that did this was compiled to minimize the overhead.
In order to compute the overhead, I had a similar loop that called
a dummy function that just returned its arguments.  Its cost was
about 0.21 seconds for the (NOUUO T) case and about 0.04 seconds
for the (NOUUO NIL) case.  I did NOT subtract these out in the following
results (but I feel they should be).

Each test run produced about 265000 conses and I had about 150000 free words.
I did a GC before each run to keep things as constant as possible.

Function	(NOUUO T)		(NOUUO NIL)

DERIV		14.879-0.875 (.213)	4.451-0.868 (.043)
DDERIV		16.250-0.856 (.198)	4.983-0.855 (.044)
FDDERIV		16.073-0.873 (.211)	4.857-0.871 (.028)
DDERIV*		18.171-1.742 (.212)	6.091-1.717 (.043)

The format of the times are
		total-gc (dummy)
where total is the total CPU time (including GC), GC is the amount used for
garbage collection and dummy is the amount of time used by the dummy loop.  I
believe the clock ticks about every 0.015 seconds which explains the
difference between the .028 and .044 dummy times (one less tick).

Explanation of results:
I believe the second and third to be slower because the property list of
the variable (whose value was the property name) had to be searched twice,
first for a function property and then for the value of the variable.
Then, for the DDERIV case, another property list had to be searched to
find the function definition.

The DDERIV* case is one in which a FUNCALL is used.  In UCILISP this
results in an extra CONS and a call to APPLY* which then does everything
done above.  As you can see, it is slower.
-------

;;;LM-2
;; 7. DERIV

(DEFUN TEST-DERIV ()
  (TIMING "DERIV" (RUN)))

;; Compiled: 23.9 seconds.

;;DERIV

D3
Elapsed	28.8
GC	18.6
CPU	10.2

Elapsed	28.5
GC	18.6
CPU	9.88

D3
1/25/84 with interrupts
CPU	15.7
GC	 6.8
;;DERIV
D2
Elapsed	79.7
GC 	39.4
CPU	40.3
PGFLT	10

D1
1/25/84 with interrupts
Elapsed	76.2
CPU	23.9
GC	52.2
;;; NIL

DERIV

cpu=31.15,elapsed=32.83,pagefaults=4428
cpu=31.14,elapsed=33.26,pagefaults=4461
cpu=31.31,elapsed=33.42,pagefaults=4469

Inline carcdr:

cpu=22.91,elapsed=24.44,pagefaults=4442
cpu=22.69,elapsed=24.63,pagefaults=4468
cpu=22.93,elapsed=25.37,pagefaults=4478
;;; SCORE dec 12, 1983
Block compiled

40.21 seconds
7.749 GC
;;; DEC780CL

	cpu + probability x gc
DERIV	14 + 1.5 x 15
;;; InterLisp Vax 780

DERIV:
←(TIME (RUN]
245030 conses
31.824 seconds
NIL
;;; PSL-20

Deriv Test.
Timing performed on DEC-20
24-Mar-84 00:19:43 .
*** Garbage collection starting
*** GC 4: time 1235 ms, 253393 recovered, 253393 free
........................................
Cpu (- GC) Time = 5.643 secs
Elapsed Time = 10.0 secs
GC Time = 1.235 secs
Load Average Before  = .8
Load Average After   = .8
Average Load Average = .8
;;; PSL-Cray
;;; Times are in milliseconds

 08:26:10 003:30.927 Deriv Test.$2 π
 08:26:42 003:41.959 Cpu (- GC) Time = 1279.98200000 secs$2 ε
 08:26:44 003:42.480 Elapsed Time = 0. secs    
 08:26:45 003:43.000 GC Time = 1877.91400000 secs$2 ε
 08:26:47 003:43.521 Load Average Before  = 0  
 08:26:49 003:44.041 Load Average After   = 0  
 08:26:50 003:44.561 Average Load Average = 0.